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PREFACE 

This  document  presents  the  results  of  actions  by  the  Optical  Systems  Group  (OSG)  of  the 
Range  Commanders  Council  (RCC)  under  Task  OS-29  to  identify  standardized  testing  methods 
for  evaluating  high-speed  digital  imagers.  This  document  defines  specific  optical  and  electronic  tests 
that  can  be  performed  to  evaluate  high-speed  digital  imagers,  guidelines  on  how  to  perform  the  tests, 
and  samples  of  actual  test  results.  Although  there  are  several  good  software  applications  available 
that  incorporate  ISO  Test  standards,  the  OSG  Working  Committee  agrees  that  the  Imatest™ 
software  should  be  accepted  by  the  RCC  working  groups  as  the  testing  standard.  Imatest™  is  a 
suite  of  programs  for  measuring  the  sharpness  and  image  quality  of  lenses,  digital  cameras,  digitized 
film  images,  and  prints  using  inexpensive,  widely  available  targets. 

The  RCC  gives  special  acknowledgement  for  production  of  this  document  to: 

Co-Author:  Mr.  William  Vining 

Associate  Member,  Optical  Systems  Group 
Redstone  Technical  Test  Center  (RTTC) 

ATTN:  TEDT-RT-F  (Bldg  7814) 

Redstone  Arsenal,  AL  35898-8052 
Telephone:  (256)  876-425 1 ,  DSN  746-425 1 
E-mail:  william.  vining@us .  army  .mil 

Co-Author:  Mr.  Eric  Husman 

Member,  Optical  Systems  Group 
P.  O.  Box  398 

ATTN:  New  Tec  Optics  Department 
White  Sands  Missile  Range,  NM  88002-0398 
Telephone:  (575)679-1873 
E-mail:  eric.husman@us.army.mil 

Please  direct  any  questions  to: 

Secretariat,  Range  Commanders  Council 
ATTN:  TEDT-WS-RCC 
100  Headquarters  Avenue 

White  Sands  Missile  Range,  New  Mexico  88002-5110 
Telephone:  (575)  678-1107,  DSN  258-1107 
E-mail:  wsmrrcc@conus.army.mil 
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ACRONYMS 


c/p 

Cycles  per  pixel 

CMOS 

Complementary  metal  oxide  semiconductor 

cy/mm 

Cycles  per  millimeter 

DIC 

Digital  Imaging  Committee 

IRIG 

Inter-range  Instrumentation  Group 

ISO 

International  Organization  for  Standardization 

LLC 

Limited  Liability  Company 

LP 

Line  pairs 

LW 

Line  width 

mm 

millimeter 

MTF 

Modulation  Transfer  Function 

NIST 

National  Institute  of  Standards  and  Technology 

OL 

Output  level 

OSG 

Optical  Systems  Group 

PH 

Picture  height 

RCC 

Range  Commanders  Council 

S/N 

Signal-to-noise-ratio 

SFR 

Spatial  Frequency  Response 

SNR 

Signal  to  noise  ratio 

u.s. 

United  States 

USAF 

United  States  Air  Force 
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CHAPTER  1 
INTRODUCTION 


1.1  Overview 

High-speed  digital  imaging  systems  are  becoming  the  dominant  means  of  photographic 
data  recording  in  private  industry  and  on  United  States  (U.S.)  government  test  facilities.  Current 
state-of-the-art,  high  resolution  sensors  are  producing  high  quality  images  that  rival,  and  in  some 
cases,  surpass  film  imagery.  Media  distribution  to  customers  occurs  now  in  a  matter  of  hours 
compared  to  sometimes  days  for  film-based  systems.  Conventional  computer  hard  drives  and 
displays  are  the  only  things  needed  to  view  test  images.  Desktop  photogrammetric  data 
reduction  algorithms  are  available  that  eliminate  painstaking  procedures  to  extract  image  metrics. 
All  of  these  improvements  have  had  an  impact  on  camera  manufacturers  to  produce  better 
imaging  products  for  use  on  the  test  Ranges. 

One  of  the  primary  challenges  for  users  of  high-speed  imaging  systems  is  to  determine 
which  system  will  best  suit  his  needs.  Camera  performance  specifications  issued  by 
manufacturers  can  be  confusing  and  may  not  be  based  upon  common  performance  standards. 

The  scope  of  this  task  is  to  identify  and  recommend  available  electro-optical  tests  and  procedures 
that  comply  with  standards  published  by  the  International  Organization  for  Standardization 
(ISO);  these  tests  and  procedures  can  be  used  to  quantify  camera  perfonnance  and  to  make 
equitable  comparisons  from  different  manufacturers.  A  synopsis  of  each  test  is  included  along 
with  references  and  links  to  detailed  infonnation  about  the  standards.  This  document 
concentrates  on  the  two  most  important  system  performance  specifications: 

a.  Resolution,  or  image  sharpness 

b.  Sensor  sensitivity  or  ISO 

1.2  Background 

Over  the  past  several  years,  all  U.  S.  Government  test  facilities  have  been  undergoing  a 
methodical  and  purposeful  transformation  from  the  use  of  the  older  mechanical  film-based 
optical  data  recording  to  electro-optical  high-speed  digital  recording  systems.  High  maintenance 
costs  for  the  older  camera  systems,  elimination  of  instrumentation  film  stocks,  and 
environmental  concerns  over  chemicals  to  process  what  film  is  available,  have  prompted 
industry  to  design  and  manufacture  solid-state  high-speed  photographic  instrumentation  to 
replace  legacy  film  systems. 

Film  cameras  have  several  independent  operational  parameters  that  limit  overall  system 
recording  performance.  Contributing  to  the  limits  of  what  film  systems  can  do  include  the 
following: 

a.  The  mechanical  drive  mechanism  that  physically  moves  the  film  through  the  camera 

b.  The  limiting  film  emulsion  speed,  grain  and  resolution 

c.  The  physical  size  weight  and  power  requirements 
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Designers  of  high-speed  digital  recording  systems  have  been  challenged  to  not  only 
replicate  film  systems  performance,  they  have  also  been  challenged  to  improve  performance  over 
film  systems.  Complementary  metal  oxide  semiconductor  (CMOS)  sensors  that  incorporate 
small,  light  sensitive  pixels  result  in  large  area,  high  resolution  imaging  devices  that  rival  the 
quality  of  instrumentation  film  emulsions.  Images  can  be  previewed  prior  to  recording  to  ensure 
that  lighting,  contrast,  focus,  and  timing  are  all  acceptable  for  data  reduction. 

Rapid  advances  in  high-speed  digital  imaging  technology  parallels  the  technology 
advances  in  the  solid-state  computer  industry.  This  is  not  surprising  due  to  increased 
requirements  of  imaging  systems  for  data  transfer  speed  and  storage  capacity.  End  users  are 
requesting  longer  recording  times  for  longer  missions  which  require  high  data  transfer  rates  to 
solid-state  recording  devices. 

There  is  an  increasing  availability  of  high-speed  digital  imaging  systems  on  the  market 
today  and  all  manufacturers  are  eager  to  emphasize  the  most  outstanding  operational 
specifications  of  their  products.  Sensor  speed,  sensor  resolution,  color  reproduction,  data 
download  speed,  timing  accuracy,  grayscale  depth,  and  dynamic  range  are  some  of  the  more 
important  performance  specifications  the  end  users  use  to  evaluate  systems  in  order  to  determine 
if  any  one  specific  imaging  system  will  meet  mission  requirements. 

One  potential  problem  with  using  the  manufacturer  specification  sheet  is  that  there  is  no 
way  for  the  end  user  to  know  exactly  how  that  particular  specification  was  measured. 

Companies  may  quote  sensor  resolution  in  number  of  pixels  but  may  not  directly  correspond  to  a 
resulting  quality  image  or  edge  sharpness.  Camera  sensor  speeds  or  ISO  specifications  are 
quoted  by  manufacturers  with  little  or  no  test  verification  to  back  up  the  numbers.  Different 
cameras  use  different  schemes  to  time  stamp  Inter-range  Instrumentation  Group  (IRIG)  on  their 
images  with  no  explanation  of  how  the  accuracies  were  actually  measured. 

Comparing  performance  specifications  among  camera  systems  using  only  manufacturers 
issued  data  without  substantiating  test  reports  could  result  in  the  end  user  choosing  a  system  that 
may  not  be  the  best  for  meeting  his  test  requirements.  Having  a  program  of  standard  testing 
protocols  that  can  be  used  to  accurately  evaluate  camera  performance  will  allow  users  to  verify 
or  disprove  manufacturers’  claims  and  to  objectively  compare  products  of  different 
manufacturers  to  one  another. 

Users  of  high-speed  imaging  systems  can  leverage  upon  standards  that  have  been  adopted 
to  evaluate  electronic  still  camera  performance  to  verify  operational  specifications  of  their  high¬ 
speed  imagers.  Well  defined  test  procedures  for  measuring  sensor  sensitivity  (ISO)  and  image 
sharpness  using  the  Modulation  Transfer  Function  (MTF)  are  presented  here. 

1.3  Existing  Standards 

The  International  Organization  for  Standardization  (ISO)  document  number 
ISO  12233:2000  describes  detailed  testing  methods  to  perform  resolution  measurements  for 
digital  still  cameras  (see  Reference  5a).  These  resolution  measurements  can  be  made  by 
analyzing  images  of  targets  on  a  specially  designed  test  chart  defined  in  the  Standard.  Visual 
resolution,  limiting  resolution,  and  spatial  frequency  response  in  terms  of  the  MTF  of  the  camera 
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lens,  sensor,  and  in-camera  imaging  processing,  are  measurable.  The  test  procedures  presented 
in  the  ISO  Standard  uses  a  diagnostic  algorithm  to  analyze  digital  image  data  of  the  specific 
slanted  edge  camera  target. 

The  ISO  12232  describes  detailed  testing  and  analytical  methods  to  determine  the  ISO 
speed  rating  for  a  camera  sensor  (Reference  5b).  The  test  procedures  presented  in  the  Standard 
use  a  calibrated  light  source  to  calculate  the  illumination  at  the  camera  sensor  plane  and  a 
computer  diagnostic  tool  to  calculate  the  speed  ratio  at  two  measured  signal  to  noise  ratios 
(SNR)s.  The  two  SNRs  of  interest  are  SNR  =10  and  SNR  =  40. 


NOTE 


1 .  There  is  no  goal  to  include  either  of  these  ISO  Standards  in  their 
entirety  in  the  body  of  this  document.  The  Standards  can  be  purchased 
from  several  sources  and  the  instructions  for  making  the  measurements 
are  well  written.  There  are  also  private  companies  that  can  evaluate 
camera  systems  based  upon  the  measurement  techniques  discussed  in 
the  Standards. 

2.  The  goal  of  this  task  is  to  identify  available  testing  procedures  and 
software  applications  that  are  compliant  with  ISO  12232  and  12233 
that  can  readily  be  adopted  by  the  user  to  thoroughly  evaluate  camera 
systems  at  the  user’s  own  facility. 
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CHAPTER  2 
TESTING  METHODS 


2.1  Resolution  Testing 

Image  crispness,  or  sharpness,  is  a  measure  of  image  quality  or  how  much  object  detail  is 
transferred  to  the  image  by  the  camera  sensor  and  lens.  This  measure  is  a  very  subjective  of 
measure  of  perceived  image  quality  since  image  sharpness  is  dependent  upon  viewing  distance. 
A  very  large  image  with  very  low  resolution,  such  as  a  billboard  sign,  when  viewed  from  a  long 
distance,  can  be  perceived  as  sharp.  The  two  primary  contributors  to  image  sharpness  are  edge 
acutance  and  spatial  resolution.  High  edge  acutance  translates  to  rapid  edge  transition  in  the 
image  or  clearly  defined  borders  and  distinct  boundaries  between  lighter  and  darker  areas. 

Lower  acutance  results  in  grey  or  soft  border  areas  when  transitioning  across  an  edge.  One  way 
to  quantify  edge  acutance  or  sharpness  is  to  use  the  rise  distance  of  the  edge  transition.  In  the 
recommended  test  procedure  the  rise  distance  is  the  physical  distance  on  the  sensor  in  pixels  or 
millimeters  (mm)  required  to  transition  from  10  percent  to  90  percent  of  the  final  pixel  level. 

The  definition,  or  object  detail,  that  is  transferred  to  the  image  is  defined  by  the  spatial 
resolution  of  the  image.  Higher  spatial  resolution  results  in  more  image  detail.  This  spatial 
resolution  is  detennined  by  measuring  how  well  the  camera  lens  and  sensor  transfers  a  series  of 
black  and  white  bar  targets  of  a  certain  density  or  frequency  in  object  space  into  image  space. 
The  overall  image  quality  and  detail  also  depend  upon  how  well  the  object  space  contrast  is 
transferred  into  the  image  plane.  A  subjective  concept  such  as  image  quality  can  be  quantified 
by  measuring  how  well  a  camera  system  transfers  spatial  detail  and  contrast  from  object  space  to 
image  space.  A  “best”  image  is  one  with  the  greatest  possible  contrast  and  the  highest  spatial 
detail  over  the  largest  percent  of  the  image  plane. 

2.1.1  Testing  with  bar  charts.  Historically,  resolution  tests  have  been  conducted  by 
photographing  a  standard  bar  chart  such  as  the  1951  United  States  Air  Force  (USAF)  chart  or  the 
newly  designed  targets  in  the  ISO  12233  chart.  Figure  2H_  illustrates  the  1951  USAF  chart  and 
Figure  2^2  illustrates  the  newer  ISO  12233  chart.  The  image  would  then  scanned  or  printed  on 
film  and  examined  to  determine  the  highest  spatial  frequency  where  image  detail  vanishes  or 
where  the  highest  spatial  frequency  where  a  discernable  pattern  is  visible.  Using  these  types  of 
charts  and  then  reproducing  the  image  for  visual  examination  is  a  very  subjective  method.  This 
limiting  spatial  frequency  usually  occurs  where  the  contrast  is  around  5  percent  and  does  not 
generally  represent  usable  images  for  photogrammetric  data  reduction. 

A  generally  accepted  level  of  image  quality  is  where  the  modulation  is  50  percent  of  the 
low  frequency  value  or  50  percent  of  the  peak  modulation  value  where  image  detail  is  very  good 
(Reference  5c). 
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Figure  2-1.  1951  USAF  test  target. 


Drawn  by  Stephen  H.Westin  ecornell  University  This  test  chart  isfor  use  with  ISO  12233  Photography  -  Electronic  still  picture  cameras  -  Resolution  measurements  Chart  Serial  No. _ Printed  by. 


Figure  2-2.  ISO  12233  Test  resolution  test  target. 
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2.1.2  Testing  With  a  Slant  Edge.  Instead  of  photographing  a  series  of  bar  targets  to  determine 
the  spatial  frequency  response  of  a  system,  a  photograph  of  a  slanted  black/white  edge  can  be 
analyzed  to  arrive  at  the  Modulation  Transfer  Function  (MTF)  curve.  According  to 
Reference  5d, 

“The  modulation  transfer  function  is  the  Fourier  transform  of  the  impulse  response.  The 
impulse  function  is  the  response  to  a  narrow  line  which  in  turn  is  the  derivative  of  the 
edge  response  which  is  the  intensity  distribution  in  the  image  of  an  edge.” 

An  image  of  a  slant  edge  from  the  bar  target  is  shown  at  Figure  2-3. 


Figure  2-3.  Image  of  a  slant  edge  from  a  bar  target. 

The  mathematical  function  that  relates  image  contrast  and  spatial  detail  is  the  MTF  or 
spatial  frequency  response.  When  the  MTF  is  plotted,  spatial  frequency  in  line  pairs  per 
millimeter  (lp/mm)  or  cycles/millimeter  (cy/mm)  are  displayed  on  a  logarithmetic  (log)  scale  on 
the  x-axis  and  the  percent  of  contrast  modulation  is  displayed  on  a  log  scale  on  the  y-axis. 

The  ISO  12233  Standard  describes  in  detail  how  the  algorithm  analyzes  the  edge  image 
and  calculates  the  derivative  of  the  edge  response  to  produce  the  spatial  frequency  response  of 
the  camera  and  lens.  In  practice,  a  single  image  of  the  black/white  slant  edge  target  is  taken  with 
the  camera  under  test  and  a  software  application  used  as  a  plug-in  module  to  an  existing  photo 
analysis  program  is  utilized  to  analyze  the  edge  in  the  image.  One  goal  of  preparing  this 
document  is  to  identify  available  applications  that  are  based  upon  and  are  in  compliance  with  the 
algorithms  presented  in  ISO  12233. 

2.2  Sensor  Sensitivity  Testing 

Another  important  performance  characteristic  for  a  high-speed  digital  camera  is  the 
sensitivity  of  the  camera  sensor.  This  sensor  sensitivity  determines  how  well  the  camera 
performs  in  low  light  situations,  with  a  fast  shutter  (short  exposure)  or  behind  long  lenses  with 
higher  f/numbers.  This  sensitivity,  or  speed  value,  is  a  number  calculated  from  an  exposure  at 
the  sensor  that  produces  a  predefined  signal  output.  The  ISO  Standard  12232  stipulates  that  a 
digital  camera  produce  a  specific  signal-to-noise-ratio  (SNR)  as  a  result  of  a  specific  illumination 
on  the  sensor  to  achieve  a  certain  ISO  value. 

The  speed  rating  for  a  digital  camera  should  be  comparable  to  those  defined  by  Standard 
ISO:5800: 1987  which  specifies  how  to  calculate  film  speed  (Reference  5e).  The  film  speed  is 
determined  from  a  plot  of  optical  density  of  the  film  (D)  versus  log  of  exposure  in  lux-seconds, 


2-3 


Standard  Testing  Protocols  for  High  Speed  Digital  Imagers,  RCC  Document  463-08,  October  2008 


also  known  as  the  Hurter-Driffield  or  “D  log  E”  curve.  Figure  2-4  illustrates  a  representative 
characteristic  curve  for  film.  The  regions  of  interest  on  the  curve  are: 

a.  The  underexposed  area  or  “base  +  fog” 

b.  The  toe  area 

c.  The  linear  region 

d.  The  shoulder 

e.  The  over  exposed  or  saturation  region 

The  minimum  exposure  (Emin)is  said  to  start  at  the  point  at  which  the  density  (D)  rises 
to  0.1  above  the  base  +  fog  level;  the  exposure  (E)  at  which  that  occurs  is  the  minimum 
exposure.  Dividing  that  value  into  0.8  gives  the  speed  of  the  fdm. 

S=0. 8/Emin  (Eq.  2-1) 


Figure  2-4.  Representative  characteristic  curve  for  film. 


This  works  for  film  because  film  is  a  threshold  detector.  In  order  to  increase  the 
sensitivity  of  film,  designers  must  increase  the  grain  size,  but  this  reduces  the  system  resolution 
performance.  Solid-state  sensors  are  linear  detectors,  so  a  camera  manufacturer  can  always 
increase  the  gain  of  the  system  to  obtain  better  signal.  Unfortunately,  this  increase  in  system 
gain  also  increases  the  system  noise,  so  this  technique  yields  an  improvement  in  paper 
specification  but  not  in  field  perfonnance.  This  is  analogous  to  increasing  gain  size,  but  it  is 
unlikely  that  a  camera  manufacturer  would  explicitly  report  the  corresponding  reduction  in 
spatial  resolution.  These  problems  are  mitigated  by  using  the  sensor’s  SNR  as  the  reported 
metric. 
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The  ISO  Standard  12232  describes  two  methods  of  calculating  sensor  speed  in  detail:  the 
saturation-based  speed  and  the  noise-based  speed.  Actually  two  ISO  noise  based  ratings  are 
recommended;  one  that  corresponds  to  an  exposure  that  produces  a  first  “acceptable”  image 
(SNR=10)  and  one  that  corresponds  to  an  exposure  that  produces  a  first  “excellent”  image 
(SNR=40).  The  Standard  defines  the  following: 


S/Nx 


ti-g(H) 
<j{Dh ) 


(Eq.  2-2) 


Where: 

S/Nx  is  10  (corresponding  to  the  first  acceptable  image)  or  40  (corresponding  to 
the  first  excellent  image). 

H  is  the  photometric  exposure,  in  lux-seconds.  The  source  intensity  must  be 
known  or  measured  by  National  Institute  of  Standards  and  Technology 
(NIST)-traceable  means;  it  is  combined  with  the  exposure  and  f/#  to 
calculate  the  focal  plane  exposure. 

g(H)  is  the  rate  of  change  in  the  output  level  divided  by  the  rate  of  change  in  the 
input  level.  This  must  be  calculated  from  values  in  successive  test  frames. 

a  (Dh)  is  the  standard  deviation  of  the  monochrome  output  level  values  taken 
from  a  standard  (64  x  64)  pixel  area.  This  must  be  calculated  from  each 
test  frame. 

Dh  is  the  monochrome  output  level. 

A  number  of  images  captured  over  the  entire  luminance  range  of  the  sensor  must  be 
analyzed  to  determine  the  SNR  for  a  camera  sensor.  A  calibrated  light  source  must  be  used  so 
that  the  source  intensity  will  be  known.  The  light  source  may  be  used  with  an  integrating  sphere 
to  produce  a  uniform  light  source.  The  intensity  is  typically  varied  with  a  variable  aperture  so 
that  the  lamp  color  temperature  remains  constant.  In  practice,  the  analysis  of  the  images  must  be 
performed  with  third  party  software  applications.  It  is  one  goal  of  this  document  to  identify 
those  applications  that  are  based  upon  and  in  compliance  with  the  analysis  methods  presented  in 
the  ISO  12232  Standard 
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CHAPTER  3 

COMPLIANT  TEST  PROCEDURES  AND  ANALYSIS  TOOLS 

After  exhaustive  surveys  in  the  open  literature  and  internet  searches,  the  Range 
Commanders  Council  (RCC)  digital  imaging  committee  (DIC)  identified  one  primary  software 
application  that  is  compliant  with  the  existing  ISO  Standard  for  resolution  testing.  This 
application  is  the  Imatest™  software  package  available  through: 

Imatest™,  LLC 
3478  16th  Circle 
Boulder,  CO  80304 

http://www.imatest.com/docs/contact.html 

The  Imatest™  software  is  not  the  only  application  for  sensitivity  measurement.  The  user 
must  also  have  an  integrating  sphere  and  third  party  software  for  analysis.  The  sphere  and 
software  can  be  purchased  as  a  unit  or  the  user  can  purchase  just  the  source  and  then  download 
recommended  analysis  software. 

3.1  Imatest™  Resolution  Testing  Application 


The  Imatest™  software  application  is  available  for  purchase  and  download  at 
http://www.imatest.com/docs/contact.html.  The  computer  code  that  calculates  the  MTF/ 
Special  Function  Register  (SFR)  are  from  a  Matlab  program  written  by  Mr.  Peter  Bums  to 
provide  the  means  to  make  the  ISO  Standard  functional.  The  original  Matlab  code  is  available 
on  the  I3A  tools  download  page  (Reference  5f). 

Many  companies  from  several  different  optical  industries  have  adopted  Imatest™  as  their 
standard  testing  protocol  for  resolution  and  MTF/SFR  testing.  This  software  application  is  a 
very  powerful  tool  that  provides  a  complete  range  of  analysis  programs  for  other  image  quality 
characteristics  other  than  resolution. 

There  is  no  intent  here  to  go  into  full  discussion  of  the  functionality  of  Imatest™.  The 
software  provides  very  detailed  instructions  for  the  user  to  measure  system  resolution  and  MTF. 
One  brief  example  is  presented  here. 

Figure  2^2  shows  the  full  resolution  test  chart  from  ISO  Standard  12232  documentation. 

A  small  area  of  one  slant  edge  section  is  used  to  determine  the  SFR  using  the  Imatest™  software. 
Figure  243  illustrates  an  enlarged  region  of  interest  of  the  slant  edge  that  the  analysis  code  uses  to 
do  the  calculation.  Figure  30_  is  an  actual  calculated  MTF  in  line  widths  per  mm. 

There  is  a  lot  of  information  displayed  from  the  Imatest™  test  results  (Figure  3-1).  The 
first  is  the  title  of  the  file  from  the  calculation,  in  this  case  “Sample  image  from  high-speed 
camera.”  The  left  side  of  the  upper  chart  denotes  that  a  vertical  edge  was  photographed  to 
calculate  a  horizontal  edge  profile.  The  actual  image  height  and  width  in  pixels  is  given  that  was 
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used  for  the  calculations  along  with  the  total  number  of  pixels  in  the  camera  sensor.  The  default 
gamma  of  0.5  is  the  estimate  of  camera  gamma  used  to  linearize  the  image. 

The  black  text  on  the  right  side  of  the  upper  chart  denotes  the  unsharpened  10  percent-90 
percent  edge  rise  distance  in  pixels  as  outlined  in  paragraph  2J_  (Resolution  Testing). 

The  lower  plot  charts  the  spatial  frequency  response  of  the  sensor  and  lens  combination, 
normalized  on  the  X  axis  versus  line  widths  per  picture  height  (LW/PH)  on  the  Y  axis. 

Another  important  result  also  given  on  the  right  side  of  the  MTF  plot  is  the  spatial 
frequency  at  the  50  percent  modulation  point.  This  50  percent  modulation  relates  back  to 
perceived  image  sharpness.  It  is  given  in  cycles  per  pixel  (c/p)  and  LW/PH. 


Sample  image  from  high  speed  camera 
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F igure  3  - 1 .  Imatest™  test  results . 
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3.2  Sensitivity  Testing  Applications 

Paragraph  22  stated  that  a  calibrated  light  source  is  required  to  determine  camera  sensor 
sensitivity.  The  system  should  include  an  integrating  sphere  controller  to  vary  the  intensity  so 
that  measurements  are  repeatable  and  deterministic.  The  system  should  also  include  means  to 
measure  the  intensity  in  photometric  units  (lux).  In  practice,  the  f/#  and  exposure  are  to  be  kept 
at  the  same  level  and  the  intensity  is  varied  with  the  integrating  sphere  controller.  The  reported 
light  intensities  are  to  be  recorded  for  each  test. 

Most  of  the  camera  interface  software  includes  a  tool  for  reporting  the  pixel  gray  scale 
values.  For  an  8-bit  system,  these  will  vary  from  0  to  255.  Because  the  S/N  (i.e.  SNR)  is 
meaningless  for  0  noise,  images  are  not  captured  when  the  pixels  show  values  of  0  or  255  since 
this  means  that  the  light  has  either  not  risen  above  the  noise  floor  or  that  has  risen  above  the 
saturation  level.  Effort  to  capture  test  images  is  concentrated  near  the  noise  floor  end  of  the 
scale,  as  not  all  cameras  are  able  to  obtain  S/N  of  40  before  going  into  saturation. 

A  software  tool  is  recommended  for  measuring  the  signal  (referred  to  in  the  ISO  Standard 
as  Output  level  or  OL)  and  the  noise  (a  (DH)).  The  measurements  must  be  made  on  a  square 
array  of  pixels,  64  on  edge,  or  4096  pixels.  The  output  level  is  simply  the  average  of  the  gray 
scale  values  in  a  monochrome  system,  or  the  weighted  average  in  a  color  camera.  The  noise  is 
the  standard  deviation  of  those  4096  points.  Several  image  processing  packages  are  capable  of 
making  these  calculations.  ImageJ,  a  Java-based  software  packaged  developed  by  the  National 
Institutes  of  Health  (http://rsb.info.nih.gov/ij/),  has  been  used  for  this  purpose. 

The  final  portion  of  the  calculation  requires  the  user  to  calculate  the  gain,  g(H),  from  the 
above  measurements  and  calculations.  The  exposures  corresponding  to  S/N  =10  and  S/N  =  40 
are  unknown  prior  to  the  test,  so  it  is  unlikely  the  experimenter  will  actually  capture  test  images 
corresponding  to  these  values.  Therefore,  the  corresponding  exposure  will  have  to  be  found  by 
interpolation.  Testing  and  measurement  done  at  White  Sands  Missile  Range  completed  the  last 
step  to  this  process  using  custom  Excel  spreadsheets.  The  process  is  that  inputs  to  the 
spreadsheet  include  the  measured  light  level,  f/#,  and  exposure  level,  all  of  which  are  combined 
to  yield  the  exposure  (lux-seconds).  The  output  levels  and  standard  deviations  are  used  to  find 
the  gain  and  S/N.  The  S/N  is  plotted  against  exposure  on  a  log-linear  graph  (S/N  is  linear, 
exposure  is  logarithmic,  typically  encompassing  the  range  from  1 0  4  to  10°  but  adjusted  as 
necessary). 

The  calculations  to  be  used  are  explicitly  written  out  in  ISO  12232.  The  focal  plane 
exposure  (H)  in  Lux-seconds  is  given  by  Equation  3-1,  which  is  derived  from  basic  principles  of 
radiometry  (see,  for  example,  Reference  5g). 

H  =  TLens  *  COS4(0)  *  7t/4  *  (L  *  t)/A2  +  Hpiare  (Eq.  3-1) 
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When  assumptions  are  made  about  TLens,  theta,  and  flare  values,  Equation  3-1  reduces  to: 

H  =  .65  *  (L  *  t)/(A2)  (Eq.  3-2) 

For  Equation  3-1  and  Equation  3-2,  the  following  applies: 

H  is  the  focal  plane  exposure  in  Lux-seconds. 

TEens  is  lens  transmission. 

L  is  brightness  in  candelas  per  square  meter,  measured  from  the  integrating 
sphere  source. 

t  is  the  integration  time. 

A  is  the  f/#. 

9  is  the  off-axis  angle  to  the  source. 

Hpiare  is  the  contribution  to  H  from  lens  flare. 

The  incremental  gain  is  described  as  follows  in  the  ISO  Standard: 

“Determine  the  incremental  gain  of  the  system  at  each  exposure, 
g(Hj),  by  averaging  the  change  in  output  level  divided  by  the 
change  in  exposure  when  going  from  the  exposure  immediately 
below  the  exposure  to  it,  with  the  change  in  output  level  divided  by 
the  change  in  exposure  when  going  from  the  exposure  to  exposure 
immediately  above  it.” 

This  (incremental  gain)  relationship  is  given  by: 

g(Hj)  =  OL(Hj)  -  OUHj  -  SH,  j)  /  2 8th,  j  +  OL(Hj )  +  5Hj, *  -  OL(Hj) /  2 SHj, *  (Eq.  3-3) 

Where 


Hi,  Hj,  and  Fh  are  the  exposures  of  the  previous,  current,  and  next  frame, 
respectively 

OL(Hx)  is  the  output  level  corresponding  to  Hx  (average  grey  scale  value 

for  the  64  x  64  block) 

5Hij  is  the  change  in  exposure  between  Hj  and  H, 

Finally,  the  sensor  speed  at  any  one  S/N  value  is  calculated  from  equation  7  in  the  ISO 
Standard  as: 


ISOx  =  10/Hs/n(x) 


(Eq.  3-4) 
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CHAPTER  4 
CONCLUSION 

The  objective  of  the  Range  Commanders  Council  (RCC)  task  which  resulted  in  the 
production  of  this  document  was  to  identify  optical  test  procedures  that  could  be  performed  by 
the  end  user  to  quantify  two  of  the  primary  high-speed  imager  characteristics  of  resolution  and 
sensitivity.  The  requirement  was  that  the  tests  be  based  upon  and  be  compliant  with  accepted 
standards  for  resolution  (ISO  12233)  and  sensitivity  (ISO  12232).  Goals  were  that  the  test 
procedures  would  be  straightforward,  could  be  done  in  a  timely  manner  at  a  user’s  facility  to 
verify  manufacturers’  claims,  and  could  quantify  imager  performance  under  certain  lighting 
conditions  and  with  different  optics  on  the  cameras. 

Thorough  technical  surveys  of  publications,  journals,  open  literature,  and  Internet  sites 
were  performed. 

The  most  thorough  and  technically  sound  application  for  the  resolution  testing  was  the 
Imatest™  software  available  through: 

Imatest™,  LLC 
3478  16th  Circle 
Boulder,  CO  80304 

http://www.imatest.com/docs/contact.html. 

The  Imatest™  software  is  well  documented  and  is  straightforward  to  run  on  any  user 
computer.  Using  this  software,  an  image  of  a  standard  test  chart  can  be  analyzed  and  a  resulting 
graph  of  the  Modulation  Transfer  Function  (MTF)  can  be  printed.  The  image  processing 
software  automatically  accomplishes  measurements.  The  software  not  only  measures  the  spatial 
frequency  response,  it  measures  other  imager  characteristics  as  well.  The  RCC  Optical  Systems 
Group  (OSG)  recommends  that  the  Imatest™  software  be  the  testing  application  of  choice  for 
resolution  measurement. 

An  alternative  to  purchasing  the  Imatest  software  and  performing  the  testing  in  house  is 
to  task  a  commercial  testing  laboratory  to  evaluate  camera  performance.  Outsourcing  this 
service  does  have  the  advantage  that  the  user  does  not  have  to  dedicate  laboratory  testing  space, 
computing  power  and  personnel  to  perform  the  analysis.  Several  optical  companies  that  have 
expertise  in  optical  manufacturing  and  testing  were  identified  during  the  completion  of  this  task. 

Optikos  Corporation  is  one  company  that  manufactures  equipment  for  the  measurement  of 
optical  image  quality.  The  company  provides  product  testing  services  that  can  evaluate  high  speed 
digital  imager  performance.  Optikos  is  located  in  Wakefield,  MA.  The  web  address  is 
www.optikos.com. 

Another  facility  that  was  identified  was  Optical  Testing  Laboratory  in  Corvallis,  OR. 
They  provide  image  quality  evaluation  for  optical  systems  and  information  technology  systems. 
Their  contact  information  is  http://www.opticaltesting.com/Contact.htm 
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Another  option  for  the  user  (instead  of  using  the  Imatest  software  and  printing  the 
embedded  test  charts)  is  to  purchase  pre-printed  resolutions  test  charts  and  down  load  third  party 
MTF  testing  algorithms.  This  option  may  be  less  expensive  in  the  long  run,  and  purchasing 
commercially  printed  test  charts  guarantees  good  grey  scale  and  dynamic  range. 

Another  version  of  the  MTF  analysis  software  can  be  obtained  from  MITRE  Corporation 
at  http://www.mitre.org/tech/mtf/. 

MITRE  has  an  application  to  compute  the  MTF  using  a  sine  wave  target  or  the  contrast 
Transfer  Function  using  a  printed  bar  target.  They  also  have  modified  existing  algorithms  to 
compute  the  spatial  frequency  response  using  an  edge  target. 

A  source  to  purchase  the  pre-printed  resolution  targets  and  charts  is  Sine  Patters  LLC  in 
Pittsford,  NY.  Their  contact  information  is 
http://www.photonicsonline.com/storefronts/sine.html. 

The  method  for  detennining  high-speed  imager  sensor  sensitivity  did  not  result  in  such  a 
straightforward  solution.  This  measurement  requires  the  user  have  a  controllable  standard  light 
source  to  calculate  the  incident  radiation  on  the  sensor  by  knowing  the  camera  exposure  and  lens 
f/#.  A  series  of  images  recorded  at  increasing  exposure  levels  must  be  evaluated  to  measure  the 
output  level  of  and  to  calculate  the  sensor  gain  with  expressions  given  in  paragraph  3.2.  The 
exposure  levels  where  S/N  =10  and  S/N  =  40  are  the  two  primary  points  of  interest.  The  ISO  of 
the  sensor  can  then  be  calculated  at  these  two  levels. 

If  the  user  chooses  to  purchase  an  integrating  sphere  and  control  system  to  evaluate  sensor 
sensitivity  in  house,  one  source  is  Labsphere,  Inc.,  in  North  Sutton,  New  Hampshire.  Their  web 
contact  information  is  http : //www. labsphere . com/servic es . aspx#c alibration. 

The  user  can  also  outsource  the  task  of  evaluating  sensor  sensitivity  just  as  in  the  case  of 
MTF  measurement.  This  does  save  the  initial  investment  of  purchasing  the  integrating  sphere  and 
control  systems.  One  good  source  for  having  the  camera  sensor  evaluated  is  Electro  Optical 
Industries,  Inc.,  in  Santa  Barbara,  CA.  Their  contact  information  is  http://www.electro- 
optical.com/html/ 

Users  of  high-speed  imaging  systems  have  several  options  to  evaluate  imager 
performance  using  the  recommended  test  procedures  for  sensitivity  and  resolution  or  to  out 
source  the  evaluation  to  commercial  testing  facilities 
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